A EH 
State Support Team 


SLDS Data Use Standards 
Standards in Practice 


Contents 


BOCK GIOUNG sisivedeavectecesscctccsavactecccscatadesiccbececceubscseuasbicaatendecuasaubsaeacanteceovestecneiaubes 2 
elu ei ° Ce, [2 Raaeneemtnene nent pitt r Bn enanae "ooh Sele rhiee nit itn tpn nr ne}teyitn le Senenene te eh st rrenonteper 4 
Metropolitan Nashville PUbliC SCHOOIS ...............ccsscccssccssrccssccesecessscessecsseceeeeceeees 8 
Manchester, NEw HOMpsnire® cassisssssssssscsssssssssscssesessnstesecsnstenesnnstenesscstenessessens 11 
Appendix A: Data Use Standards .................ccccccsessssssssscccccccceeesesssssseccceceeeeees 20 
Appendix B: Data Use Standards Workgroup Membe!s  ...............cceeeseeeeeee 22 


Standards in Practice, July 200 


Background 


The Need 


As state and local education agencies increasingly focus on serving educators with their data systems, a common challenge has 
arisen: identifying the critical knowledge, skills, and behaviors needed by teachers and administrators to use data effectively. 
Many states are creating data literacy and data use training programs for pre- and in-service educators without a common 
foundation on which to base the content. In addition, several state education agencies and educator preparation programs have 
begun communicating about how to create a stronger alignment between pre- and in-service training for educators regarding 
data use. 


Goal and Objectives 


Goal: To increase the effective use of data by educators to support student learning and success. 
Objectives 


1. To provide a foundation for states’ development of data literacy and data use trainings. 
2. To inform and improve the articulation between pre- and in-service data training for teachers and administrators. 


Contents and Intended Audiences 


This resource contains three case studies that illustrate the essential knowledge, skills, and professional behaviors necessary to 
use data effectively being applied by educators in real-world settings. It is a companion publication to SLDS Data Use Standards: 
Knowledge, Skills, and Professional Behaviors for Effective Data Use. Specific standards from that publication are illustrated in these 
case studies and referenced in parentheses (e.g., K.7.A Question Formation). See Appendix A for a complete list of standards. 


This resource is intended to inform pre- and in-service educator training programs. The language is purposefully nontechnical 
to make it accessible for educators. 


Approach 


The content of this resource was developed by the Data Use Standards Workgroup of the Statewide Longitudinal Data Systems 
(SLDS) Grant Program. State agency staff from Hawaii and South Dakota generated the idea for the workgroup. An open 
invitation to participate was sent to all states via the SLDS listserv, and the workgroup was formed in November 2013. 


Authors 


This resource was created by the 27 members of the Data Use Standards Workgroup representing 14 states and Guam! and 
facilitated by the SLDS Grant Program State Support Team. Members represent state education agencies, local education 
agencies, regional service agencies, postsecondary institutions, and state P-20W+ (early childhood through workforce) 
longitudinal data systems. Appendix B contains the list of workgroup members. 


' California, Guam, Hawaii, Idaho, Illinois, Kansas, Montana, Nebraska, New Hampshire, New York, North Dakota, Oklahoma, South Dakota, Tennessee, 
and Washington. 


2 SLDS Data Use Standards 


Beginning of Idea Recruit and Plan Release Phase | Release Phase II 
(August 2013) (November 2013) Resource (July 2014) Resource (July 2015) 


Discuss with SST Launch of Phase | Launch Phase I] 
(Fall 2013) (January 2014) (February 2015) 


Figure 1. Timeline of the development of SLDS Data Use Standards resources. 


Suggested Citation 


Statewide Longitudinal Data Systems Grant Program. (2015). SLDS Data Use Standards: Standards in Practice. 
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This publication is available online at bitps://slds.grads360.org/#t#program|/ data-use-standards. 
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ia North Dakota 


Introduction 

Educators in North Dakota are using A+ Inquiry (see figure 2) as a framework to guide data conversations and to facilitate 
effective data use. A+ Inquiry promotes continuous awareness of a complete inquiry cycle to ensure, for any given data 

use process, that “the right context is absorbed, the right questions are asked, the right data are accumulated, accessed, and 
analyzed, the right answers are derived, the right announcements are communicated, and the right application is made” 
(Anderson, Brockel, & Kana, 2014, p. 6). This scenario applies A+ Inquiry as a lens for interpreting how an elementary teacher 
could implement data use standards to set reading goals with students, communicate progress to parents during conferences, 
and inform action to improve reading achievement. 
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Figure 2. A+ Inquiry framework. 


ABSORB 


Defining the context 


Mr. Gonzalez is a fifth-grade teacher. In October 2014, he is preparing to set a reading goal with a student named Stuart. He 

is also prepating to discuss Stuart’s reading achievement and goals with Stuart’s parents at parent-teacher conferences in the 
upcoming weeks. Stuart recently completed the fall 2014 norm-referenced standardized reading assessment and will take the 
assessment again during spring 2015. Stuart also completed the fall 2013, winter 2014, and spring 2014 standardized reading 
assessments during the previous school year. During the goal-setting meeting with Stuart and the parent-teacher conference 
with his parents, Mr. Gonzalez would like to discuss the extent to which Stuart’s reading achievement grew from fall to spring 
during the previous year, how his score compares to district average and national average scores, the goal areas that represent 
his highest and lowest levels of achievement in reading, and his Lexile reading level. However, Mr. Gonzalez does not currently 
know enough details about Stuart’s reading achievement to have meaningful, relevant conversations with him or his parents 
(K.2.D Data Context). 
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ASK 


Using questions to guide data use 


Mr. Gonzalez begins the process of learning more about Stuart’s reading achievement by asking the following questions: 


* How does Stuart’s fall 2014 scale score compare to the national average scale score for his grade level? 
* How does Stuart’s fall 2014 score compare to the district average scale score? 
* How does Stuart’s fall 2014 scale score compare to his spring 2014 scale score? 
* How does Stuart’s actual fall 2013 to spring 2014 scale score growth compare to his projected fall 2013 to spring 
2014 scale score growth? 
* How does Stuart’s spring 2014 percentile compare to his fall 2014 percentile? 
¢ What is Stuart’s projected scale score for spring 2015? 
* How many scale points is Stuart expected to grow between fall 2014 and spring 2015? 
¢ How well did Stuart perform on the fall 2014 assessment in each of the reading goal areas? 
¢ What is Stuart’s Lexile level? 
(K.1.A Question Formation; S.1.A Goals and Questions) 


ACCUMULATE 


Identifying helpful data sources 


Mr. Gonzalez determines that quantitative standardized assessment reading data accumulated during fall 2013, spring 2014, 
and fall 2014 could help answer his questions (K.7.C Types of Data). 'The required data were collected through a computerized 
adaptive assessment in alignment with administration procedures required by the test vendor (K.2.C Data Collection; $.1.C Data 
Management, S.3_A Facilitation; S.3.B Technology). 


The assessment measures academic achievement and growth in reading (K.7.D Types of Measures; S.3.C Multiple Measures). Mt. 
Gonzalez collaborated with the school counselor to coordinate data collection for each iteration of the assessment (B.3.A 
Collaborative Use). Testing accommodations were available for students with special needs; Stuart was allotted extra time to 
complete each assessment (S.3.D Modifications). Mr. Gonzalez is confident using the data to answer his questions because 
several unique studies conducted by the test vendor and external researchers provide substantial evidence of data reliability 
and validity (S.2.B Critical Evaluation; K.1.B Data Quality; B.1.A Data Quality). However, he acknowledges that the data alone are 
limited in the extent to which they provide a complete understanding of Stuart’s reading achievement (K.3.B Data Limitations). 


ACCESS 


Obtaining relevant data 


Mr. Gonzalez accesses the fall 2013, spring 2014, and fall 2014 data he needs by logging into his secure account through the 
assessment vendor’s website (K.7.F Data Sources; S.2.A Data Discovery and Data Acquisition). He navigates to the reports section 
of the website and runs the Student Progress Report. 


He specifically needs the following descriptive data: 


°  Stuart’s fall 2014 scale score 
* most recent national average grade-level scale score 
° — fall 2014 district grade-level average scale score 
°  Stuart’s spring 2014 scale score 
° Stuart’s spring 2014 percentile 
*  Stuart’s fall 2014 percentile 
°  Stuart’s fall 2014 percentile category by goal area 
*  Stuart’s spring 2015 projected scale score 
*  Stuart’s Lexile range 
(S.7.D Data Meaning). 


The report presents data in a format conducive for analysis (K.2.E Data Format). 
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ANALYZE 


Determining student progress 


Mr. Gonzalez dedicates time to analyze data in the Student Progress Report (B.3.D Prioritization). He learned how to analyze data 
in the report during a professional development workshop conducted by the assessment vendor (B.4.C Professional Development). He 
directs his attention to the Reading section of the Student Progress Report (8.5.4 Locating). He analyzes descriptive data by looking at 


* the fall 2014 columns to compare Stuart’s score to the national and district grade-level averages; 

° the fall 2013 and spring 2014 rows in the table to compare scores achieved during each term and compare the 
projected and actual fall 2013 to spring 2014 score growth; 

* the spring 2015 score projection column to determine the spring 2015 projected score and projected growth from 
fall 2014 to spring 2015; 

* the goal performance descriptors to determine performance in each goal area; and 

* the Lexile range to determine Lexile level. 

(K.2.A Types of Analysis; $.4.D Considerations; S.4.E Comparisons; S.5.D Congruency). 


ANSWER 


Responding to the questions 


Mr. Gonzalez revisits the questions posed in the Ask stage. Analysis illuminated answers to his questions (S.4.C Aligned 
Analysis). He determines that 


°  Stuart’s fall 2014 score is lower than the national grade-level norm, lower than the district grade-level mean, and 
lower than his spring 2014 score; 

° his actual fall 2013 to spring 2014 growth was higher than his projected fall 2013 to spring 2014 growth; 

° his fall 2014 percentile was lower than his spring 2014 percentile; 

° he performed at a low-average level in the Literature goal area, an average level in the Vocabulary Acquisition and 
Use goal area, and a low-average level in the Informational Text goal area; 

* his Reading scale score is projected to grow 5 points between fall 2014 and spring 2015; 

° his spring 2015 scale score is projected to be 207; and 

° his Lexile range is between 537-687. 

(S.9.C Patterns). 


ANNOUNCE 


Communicating the answers 


Mr. Gonzalez announces the answers to himself, to Stuart, and to Stuart’s parents (8.6.C Multiple Audiences). He decides that he 
will assign appropriately leveled articles to Stuart for reading assignments and will group Stuart with other students achieving 
at similar levels during relevant lessons. He keeps the printed reports in Stuart’s file in the file cabinet under his desk to ensure 
that the documents will not be viewed by someone other than Stuart or his parents (B.2.A4 Rules and Laws; K.3.D Privacy and 
Security; B.1.D Ethics; B.2.B Protection). 


Mr. Gonzalez announces the answers to his questions about Stuart’s reading achievement to Stuart during a teacher-student 
goal-setting meeting. He retrieves the Student Progress Report and Student Goal-Setting Worksheet from the assessment 
vendotr’s online reporting website to use as visual aids when communicating the answers to Stuart (8.6.4 Presentation). 


He uses the Student Progress Report to inform Stuart that 


° he is performing slightly lower than the average for other fifth-grade students in the nation and the district; 
* his score grew more than expected from fall 2013 to spring 2014, but his fall 2014 score is lower than in spring 2014; 
° he is scoring lower than more students in the nation this fall than he did last spring; 
* he is projected to grow 5 points by the following spring; and 
° he is performing at a low-average level in Literature, an average level in Vocabulary Acquisition and Use, and a low- 
average level in Informational Text. 
(B.1.B Transparency; S.6.B Explanation). 
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Mr. Gonzalez and Stuart use the Student Goal-Setting Worksheet to set a goal for Stuart to grow 5 points by spring 2015 and 
set an action plan for Stuart to read one appropriately leveled article per week based on his Lexile range (S.7.B Action Plan). 
The articles can be accessed through various websites offering Lexile-leveled literature (S.7_A Strategies). Mr. Gonzalez and 
Stuart agree to measure progress toward Stuart’s goal following subsequent standardized assessments completed during the 
winter and spring terms. 


A couple of weeks following the goal-setting meeting, Mr. Gonzalez announces the answers to Stuart’s parents during parent- 
teacher conferences (B.7.B Transparency). He uses the Student Progress Report and Student Goal-Setting Worksheet to facilitate 
a conversation about Stuart’s reading achievement, goal, and action plan (B./.B Transparency; S.6.B Explanation; K.1.G Data 
Representations). He provides Stuart’s parents with links to the websites where appropriately leveled articles can be retrieved 
online and encourages Stuart’s parents to support his goal and action plan to read one online article per week at home. 


APPLY 


Implementing the action plan 


Mr. Gonzalez, Stuart, and Stuart’s parents take action intended to improve Stuart’s reading achievement (B.4.B Improving 
Outcomes). Mr. Gonzalez assigns appropriately leveled articles to Stuart and groups Stuart with other students achieving at 
similar levels during relevant lessons. Stuart reads one appropriately leveled article per week at home based on his Lexile level. 
Stuart’s parents encourage him to implement his action plan at home and help him access appropriately leveled articles online. 


Mr. Gonzalez periodically checks in with Stuart and his parents to determine if adjustments need to be made and if progress is 
occurring (S.7.B Action Plan; $.7.A Strategies). 


AWARENESS 


Staying engaged 
Mr. Gonzalez demonstrates awareness throughout the process, ensuring that the right context is absorbed; the right questions 


are asked; the right data are accumulated, accessed, and analyzed; the right answers are derived; the right announcements are 
communicated; and the right applications are made. (S.7.B Alignment). 


NOTES 


¢ A+ Inquiry framework adapted from Anderson, N. C., Brockel, M. R., and Kana, T. E. (2014). Disciplined inquiry: 
Using the A+ Inquiry framework as a tool for eliminating data hoarding, mindless decision-making, and other 
barriers to effective ESA programming, Perspectives: A Journal of Research and Opinion About Educational Service Agencies, 20(3). 

° This scenario was informed by a Northwest Evaluation Association Measures of Academic Progress (NWEA 
MAP) reading assessment commonly administered to elementary school students in several districts throughout the 
US. Terms and acronyms specific to NWEA MAP assessments were written in generalized language to simplify 
concept interpretation for readers lacking familiarity with NWEA MAP. For more information visit www.nwea.org, 
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* Metropolitian Nashville 


Public Schools 


Introduction 

Metropolitan Nashville Public Schools (MNPS) is the 42nd-largest school district in the United States, with approximately 
87,000 students. Using Race to the Top funding, Fred Carr, the chief operating officer for MNPS, led the development of 

a data warehouse in 2011 to begin consolidating data silos that existed across the district (S.3.B Technology). Having a district 
data warehouse led to some interesting observations and questions (S.2.A Data Discovery and Data Acquisition). One interesting 
observation made by Dr. Tony Majors, chief officer of support services, was a trend among chronic absence data and other 
types of data, such as at-risk, discipline, and grades (S.4.E Comparisons). The initial observations led to a deeper data dive into 
the issue of chronic absence (S.5.C Patterns). 


After seeing the initial trends in chronic absence, a meeting was convened to investigate the trends. The driving question for 
the meeting was “What effect(s) does chronic absence have on student achievement and discipline?” (K.7.A Question Formation). 
While initially the data indicated a relationship, it was critical that more data be included and more people be involved in the 
discussion before developing an initiative around the issue (S.3.A Facilitation; B.3.B Collaborative Climate; B.3.C Outreach). 


Figure 3. MNPS representatives meet to discuss 
trends in chronic absence among students. 


After analyzing data from multiple sources (S.3.C Multiple Measures), it was determined that the attendance data needed to be 
viewed from the lens of chronic absence and not just attendance rate (S.7.A Goals and Questions; S.7.A Strategies). Attendance 
rate alone was masking some issues for several of MNPS’s lowest performing students (S.9.B Representation). With the key issue 
identified, three critical solutions were developed: 


1. Build a core team’s capacity about chronic absence 
2. Create a data warehouse attendance report to depict chronic absence data 
3. Develop training support materials 

(S.7.B Action Plan). 


BUILDING A TEAM’S CAPACITY 


Although staff within the district knew about chronic absence, there was some uncertainty about what it truly meant, 
patticularly when reporting a data point (S.7.B Algnment). MNPS reached out to Attendance Works, a national and state 
initiative that promotes better policy and practice around school attendance. After meeting with representatives from 
Attendance Works, common language defining chronic absence was adopted. MNPS adopted Attendance Works’ definition 
for chronic absence: Students are chronically absent if they are “missing 10 percent or more of school days including all 
absences, excused or unexcused or even suspensions” (Attendance Works, 2014) (S.7.D Data Meaning). 
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CREATING AN ATTENDANCE REPORT 


With a common definition for chronic absence, the MNPS Office of Support Services and Data Warehouse team collaborated 
to develop an attendance dashboard report (see figure 4) (S.7.C Data Management). It took approximately a week to develop and 
finalize the report. Currently, a dashboard report is updated every 24 hours and reflects the most recent data. In the MNPS 
data warehouse, data can be viewed at the district, cluster, school, and teacher level based on assigned security roles (B.7.D Ethics). 
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Figure 4. MNPS’ attendance dashboard report 
displays attendance data at the district, cluster, 
school, and teacher levels. 


DEVELOPING TRAINING SUPPORT MATERIALS 


A common assumption made around data use is that all educators understand the data. However, a national study of districts 
known for strong data use found teachers showed difficulty with posing questions and comprehending and interpreting data, 
with teachers correctly interpreting given data in only 48 percent of instances (U.S. Department of Education Office of 
Planning, Evaluation and Policy Development, 2009). To address this issue, MNPS’s business intelligence coordinator, Dr. 
Margie Johnson, facilitated the development of various training support materials (B.3.B Collaborative Climate). 


The training support material development process began by conducting three focus groups—one each at an elementary 
school, a middle school, and a high school. The focus group participants were asked to assess themselves using Attendance 
Works’ Self Assessment tool (see figure 5) (B.3_A Collaborative Use; B.3.C Outreach). 


Does Attendance Really Count in Our School? 


A Tool for Self Assessment - (Revised June 2012) 
Key Element Strength OKfor Couldbe Urgent Don’t | How do you know? 
Now Better Gap Know 

ft. Attendance is accurately taken and entered daily into the 
district data system in all classrooms. 

b. Our school has a clear and widely understood attendance 
policy. 

B. A team including the site administrator reviews attendance 
data on a regular basis to Identify chronically absent students 
and monitor attendance patterns by grade, student sub- 
groups and classroom. 

fi. Our school offers such rich and engaging learning activities 

that students do not want to miss school. 

Our school recognizes and appreciates good and improved 

attendance. 

6. Our school informs parents about the importance of 
attendance and encourages parents to help each other get 
their children to school. 

7. Our school identifies and reaches out to chronically absent 
students and their families in a caring manner to see how 
attendance could be improved. 

8. Our school partners with community agencies that offer 

resources (pre-k, afterschool, health services, volunteer 

mentors, transportation) that can help engage students & 
their families and remove barriers to getting to school. 

fp. __ Individual learning plans are developed for high-risk students 
and address poor attendance along with low academic 
performance. 

ft0. Our school discipline policy and practice ensure students do 
not miss instruction due to suspensions for non-violent 
behaviors. 

[11. Our strategies for supporting student attendance are 
reflected in our school improvement plan. 

(Note: Chronic Absence is missing=> 10% of school over a year including any type of absence; Satisfactory Attendance is missing=< 5% of school over a year) 


school Name:. 


Figure 5. Focus group participants completed a data use 
self-assessment developed by Attendance Works. 


Standards in Practice, July 2010 q 


om Attendance Dashboard After multiple datasets from the focus groups were analyzed, two trends emerged (S.3.C 

” ==— | Maltiple Measures; $.5.C Patterns): the need for a standardized, documented process and the 
need for a communication plan. MNPS data analyst Sarah Vinson worked with the schools 
to document the process (K.7.B Data Quality; K.2.E Data Format).'This documentation was 
shared in a meeting at the central office to finalize the process (B.3.B Collaborative Climate). 


Once the team agreed upon the documentation, a communication plan was developed. 
The plan included the creation of an Attendance Toolkit with the standardized process, a 
Quick Reference Guide, and an Attendance Data Guide (see figure 6) (Rankin, 2015) to be 
Be cipetpermerered , disseminated across the district (S.6.C Muluple Audiences). One theme that emerged during 

the development of the plan was the word “care.” The team used that theme to name the 
initiative C.A.R.E. (Connecting Attendance to Results in Education). 


The best part of the data-informed initiative is the ability to monitor its progress. The 
Figure 6. The Attendance _ initiative launched in August 2014. By October 2015, MNPS saw a 5 percent decrease in 


Data Guide is part of chronic absence across the district (B.4.B Improving Outcomes). 
MNPS's Attendance Toolkit. 
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y Manchester, New Hampsnire 


Introduction 

The Manchester School District is the largest school district in New Hampshire, with 14 elementary schools, four middle 
schools, and four high schools. There are approximately 1,100 students in each grade from kindergarten through 12th grade. 
The district has battled budget reductions and administrator turnover. A large percentage of students are not meeting state- 
measured proficiency in math and English language arts. 


For many years the data showed approximately 60 percent of students were not proficient in math, reading, and writing, as measured 
by the state assessment. The trend over the past several years was not showing improvement. In fact, in some subjects and 
grades fewer than half of students demonstrated proficiency, with some rates as low as 32 percent (see figure 7). 
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Grade Content Area 2009-2010 2010-2011 2011-2012 2012-2013 2013-2014 
N Yo N % N % N % N % 

Reading 1039 64 1100 61 1029 64 1069 61 1093 56 

. Mathematics 1042 62 1116 60 1039 59 1079 54 1102 48 
4 Reading 1057 58 1009 57 1088 60 995 59 1063 55 
Mathematics 1065 62 1021 59 1097 58 1003 58 1070 55 

Reading 1078 65 1043 62 991 54 1070 59 984 60 

5 Mathematics 1078 65 1060 56 996 57 1083 55 991 50 
Writing O 1049 41 990 37 1070 42 984 42 

& Reading 1136 55 1072 62 1035 59 1004 58 1083 58 
Mathematics 1149 54 1080 55 1045 52 1006 51 1087 50 

Reading 1116 53 1136 48 1047 58 1034 54 990 51 

* Mathematics 1126 42 1154 43 1064 49 1041 50 1002 44 
8 Reading 1106 53 1076 59 1136 64 1018 64 965 58 
Mathematics 1113 39 1093 41 1151 46 1024 51 976 44 

Writing te) 1083 43 1134 40 1018 52 965 32 


Figure 7. State summative assessments show student proficiency in math, reading, and writing. 


In summer 2014, the district administration in Manchester launched a targeted effort to use data to inform instruction, with 

a goal of improving student performance. The administration approached the New Hampshire Department of Education 
(DOE) and enlisted SLDS data coaches to help create a culture and practice focused on effective data use. Initial work 

was focused on kindergarten through eighth grades. Although the district had a subset of schools identified by the state as 
‘priority’ schools, the superintendent felt strongly that support should be provided to all schools in the district. The use of data 
to inform instruction was considered valuable for all schools, not only those with the lowest proficiency. 
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Through a collaborative discourse, the data coaches and the Manchester Leadership Team talked about the data needs and 
current data climate in the district. These discussions answered many questions, but also unearthed more. 


Initial meetings were held during the summer, and a plan was developed to create local assessments and improve existing 
assessments with a consistent approach throughout the district. District officials developed templates to define the assessment 
schedule, identified key personnel for data teams, discussed a communication plan, and identified gaps for further discussion 
(see figure 8, next page). The work was ready to begin. 


Key Knowledge, Skills, and Professional Behaviors were identified as part of the initial definition of the work. 


Knowledge. The initial work centered on understanding the question that Manchester wanted to answer (K.7.A Question 
Formation) and what data sources (K.1.F Data Sources) already existed within the schools. The leadership needed to determine 
what types of measurements (K.7.D Types of Measures) would be used to inform instruction. Questions about how to collect 
the data (K.2.C Data Collection) and how to analyze the data were discussed (K.2.A Types of Analysis). The Leadership Team 
decided to use the PerformancePLUS data analysis tool (K.2.B Data Analysis Tools). 


Skills. The initial discussions focused on planning what would be accomplished during the upcoming school year (5.7 
Planning), who would manage the work (S.7.C Data Management), what the district goals were, and what questions teachers 
would be able to answer (S.7.A Goals and Questions). A heavy emphasis was placed on communication. District officials 
discussed how the flow of information would be managed to ensure a single, clear message for multiple audiences (S.6.C 
Multiple Audiences), including the teachers and administrators. 


Professional Behaviors. It was immediately evident that there was no consistent culture across the schools (B.7.E Culture). 
The district wanted to ensure that there was an emphasis on collaboration (B.3.B Collaborative Climate) across schools. The 
importance of engaging leaders within the schools led to an initial meeting with all principals. The need for teacher training 
(B.4.C Professional Development)—specifically related to how teachers would learn about the collection and use of this data— 
was also part of the initial discussion. 


INITIAL STEPS: THE PROJECT TAKES OFF 


Administrative leaders, data coaches, and a small group of teachers who had been involved in analyzing data in prior years 
formed a district team to help build a foundation for the work that was about to begin. They met to define the initial steps and 
analyze existing data. Discussions uncovered pockets of assessment and data analysis that had been performed by educators 
in past years and could be leveraged so that the team did not have to start from scratch. It is important in data work that 
educators see the connection with previous work and current initiatives, so they do not see it as “just another new initiative.” 


The district team learned that new mid- and end-of-year math assessments had been written and administered throughout 
the elementary schools. They used an assessment inventory template for this (see figure 9, next page). A system had also 
been put in place to allow teachers to enter students’ math data into PerformancePLUS, the New Hampshire data system. 
PerformancePLUS could have been used to generate reports to monitor student growth, but only a few people in the 
Manchester district knew how to generate these reports. No system was in place to review the data after they were entered. 


While the overall district plan was being finalized, data coaches met with data teams at each school to assess their current 
capacity and to begin creating awareness and education about how to analyze data. A structured data dive methodology was 
used to walk data teams through an analysis exercise. A few key observations came out of this work: the school data teams 
had minimal experience and skills analyzing data, teams were at very different levels of data maturity from school to school, 
and there was no consistency from school to school as to what assessments students were taking. However, a consensus was 
reached that data should be used not only to analyze grade-level instruction, but also to ensure that students were proficient in 
core content. 


To guide the initial and ongoing work, a district-level leadership team was created for each content area (math and English 
language arts) at both the elementary and middle school levels. Each team included a district-level content leader and 
experienced Manchester educators working alongside a data coach. 


Although most of the district data teams had strong teacher representation, the data coaches struggled to ensure that key 
teachers representing every school in the district were identified. Not having ownership for the school-level work caused some 
bumps in the road as the work was first developed and rolled out. 
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Data ToolKit 


Fountas and 
Pinnell 


Student 
Perception 
Survey 


Mid Year Math 


Proposed Approach To Engage Data Teams 


The following is a proposed plan to help support data teams at the Priority schools. It provides a suggested sched- 
ule for the next few months with some additional recommendations for the remainder of the year. 


A few of guiding principles... 

The DOE will provide a data coach to help along the way. 

Each school should identify a data team, with a team lead. 

We need to recognize the successes and hard work of the data team and educators. 

The principal should be an active member of the data team. 

The team should be given the required time to have at least bi-weekly meetings. 

The team and principal must collaborate and receive support to implement interventions and system 
changes identified as necessary to address data driven instruction. 


Nap wN 


Suggested goals and approach for first few months. 


September: Building the data team and understanding base data. 


© Schools needs to identify data team --(including a lead, ideally not principal) 


* We will hold a district-wide (ie. 7 schools) full day meeting with the following agenda: 
© Discuss data tools/offerings 
[Mike and Karen] 
* Describe data team best practices (how to implement a data team) 
[Donna, Rob and Southside data lead] 
* Breakout and perform the data meeting protocol (data dive) 
[Rob to introduce; Donna, Karen, Mike and Rob to facilitate] 
Prerequisites 
-> Donna Crook needs to load data 
-> Principal, and full data team should attend 


October: Initiate the data team, further understand the data, identify priorities and develop the ID10 plan. 


Data teams should analyze data back at each school. 

Data teams should be meeting biweekly. 

Data teams should identify priorities. 

Create inventory of interventions resources: material resources / programs, expertise of educators in building, 
availability of resources. 

Data teams should begin developing ID10 plans. 

Plans should consider both Core and interventions, as appropriate. It should include instruction and profession- 
al development needs. 

Data coaches will attend one October meeting to facilitate |D10 plan 


[mt | oo oooo 


Figure 8. Sample approach plan. 


Data Inventory: What Do We Have? 
(Assessment, Demographic, Climate,) 


Example 


What is assessed 
i 


Fluency October 
Comprehension and May 
Accuracy 


Student perception 


of safety, respect 
and community in 
our school 


Grade Level Math 
to date 


Student Writing 
Ability 


Formative | How is the data 


Google Drive for 


Figure 9. Sample assessment inventory worksheet. 
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Figure 10. Sample assessment schedule. 


After reviewing the assessments and data, the teachers, data coaches, and math and literacy coaches determined that new 
assessments should be written that (1) are aligned with Manchester Standards, (2) are aligned with the units of study being 
taught, and (3) give more information to teachers about their students’ basic skills. 


Math 


> Math Fluency Assessments: Short (three-minute), monthly assessments for grades K-8 to test math fluency (eg,, 
addition, subtraction, multiplication, and division). 

> Math Computation Assessments: Short (15- to 20-minute), monthly assessments to measure grade-level 
computation skills. 

> End of Unit Assessments: Performed at the end of each unit to assess student learning for that unit. 

> Performance Tasks: Quarterly performance tasks to provide a comprehensive review of student learning. 


English Language Arts 


K-5 Fountas and Pinnell Benchmark Assessments. 

K-5 Summative Performance Tasks for each of three writing units. 

Grades 6-8 Pre and Post (Beginning and End of Year) Argumentative Writing Assessments. 
Grades 6-8 quarterly performance tasks for each unit. 


YY VY 


A timeline was created for a comprehensive assessment program (see figure 10). 


A plan was now in place to administer assessments, analyze student results (through school-based data teams), and make 
instructional decisions. Many details needed to be worked through, but the initial assessment work had begun. 


As the project was formed and the assessment work began, Knowledge, Skills, and Professional Behaviors were critical to 
the work and success. 


Knowledge. Through this phase of work, the questions being addressed were further defined (K.7.A Question Formation). For 
example, 


¢ Did students have core math competencies required for basic learning? 

e Were students learning the unit goals? 

° Was there a common set of writing expectations across cross-content educators? 
¢ What were students’ individual writing abilities? 


The district team helped unveil what existing types of quantitative data (K.7.C Types of Data) and measurements (K.7.D 
Types of Measures) currently existed in the schools, and it discussed the lack of teacher awareness about how assessment data 
should be used (K.2.D Data Context). The team discussed the importance of the data tools and how to take advantage of 
existing tools (K.2.B Data Analysis Tools), as well as the format in which data would be presented (K.2.E Data Format). 


Skills. As part of forming the assessment plan, the district team discussed the purpose of existing assessments and the 
meaning of the data (S.7.D Data Meaning). Some national assessments were being conducted, but the existing data were not 
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providing the information necessary to answer the questions being addressed, thereby preventing instructional analysis and 
change. The selection of assessments and the decision to develop local assessments were critical early steps (S.2.B Critical 
Evaluation; S.2.C Development of Measures). 


The team decided to collect multiple measures covering both student core knowledge and grade-level content (S.3.C Multiple 
Measures). It also discussed how to communicate the information, and the lead district administrator was selected to be 

the lead ‘cheerleader’ and communicator. However, over the course of this initial work, the team learned that frequent 
communication with the teachers was also critical. Improved communication was necessary for the ongoing work. 


Professional Behaviors. The district team discussed the history of data use in the district. It was important to understand 
the existing culture (B.7.E Cu/ture). To build support, the team stressed transparency so that teachers would understand 

the intent (B.7.B Transparency). The initial data dive sessions stressed the importance of not misrepresenting data (B.7.C 
Representation). Educators saw how they might have predetermined misconceptions of assessment results. The district team 
also discussed the required level of data access and protection (B.2.B Protection) and what relevant laws allowed (B.2.A Rules 
and Laws). The team decided to ensure that teachers had broad but appropriate access for transparency and to empower 
them to analyze results and improve instruction. In prior years, access was limited to such a degree that it prevented 
collaborative use (B.3.A Collaborative Use) and a collaborative climate (B.3.B Collaborative Climate). Reviewing the prior yeat’s 
efforts allowed for a tenor of continuous improvement (B.4 Continuous Improvement). 


PROJECT EXECUTION 


With goals in place and the data identified for collection and analysis, the detailed operational questions were now addressed 
and the implementation went forward. 


Several detailed processes were established to collect data and analyze the results. This work is an ongoing effort, with bumps 
and lessons learned along the way. It will certainly take multiple years to develop a smooth process. 


Collecting Data 


As described above, district data teams that were focused on elementary- and middle-school-level math and English language 
arts worked to create monthly and end-of-unit assessments. They used existing materials where possible to take advantage of 
prior work. They spent time identifying the contents of each assessment, agreed upon cut points that would define proficiency 
for students, and worked collaboratively to develop a common understanding of how these local assessments would be scored. 
For example, for a simple math fluency assessment, the team determined the number of addition problems, the time a student 
would have to take the assessment, and the required score to demonstrate proficiency, as well as other cut points. For the 
writing rubric, the team identified the metrics that would be addressed (e.g., Organization — Adequate progression of ideas, 
use of transitional words, phrases, or clauses usually help the reader connect and clarify ideas, conclusion connects to the 
argument, etc.), identified valid writing prompts, and determined the approach to score and cross-score the students’ work. 


The district teams created versions of the math and reading assessments for each month. For some assessments—such as the 
math fluency and computation monthly assessments—the assessments were similar each month. For others, such as end-of- 
unit assessments, the content changed each month. The team used a district Google Drive to share materials with teachers. 
Although this enabled access, the need for a more controlled, top-down, and timely communication plan remained. 


In addition to administering the test, a process was defined to populate the PerformancePLUS data system with the assessment 
results. Using this tool allows the teachers and administrators to analyze the data. PerformancePLUS also allows teachers and 
administrators to access the appropriate data. The system already included student background information (e.g., gender, race, 
class roster, and attendance), allowing for analysis across multiple factors. During the first few months of the project, the 
district data team worked to get all the assessment materials on Google Drive at least a month ahead of the timeline. Most of 
the work will likely be in place for the second year. 


With the materials in place, a series of communications and meetings were implemented to educate teachers on the assessment 
schedule and procedures. The collection of data had begun. 
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Data Analysis 


Data coaches played a primary role in creating the reports, which allowed administrators and teachers to analyze the student 
results. Often, standardized assessments are conducted at schools without the data being used to influence instruction. 


As described above, the data coaches worked with schools to perform a data dive methodology (see figure 11). This process 
helped focus the discussion on identifying how school factors and instruction could be modified to improve student outcomes. 
Participants went through a four-step process. After setting discussion norms, team members predicted perceptions of the 
student outcomes. They then analyzed data reports without discussing causation (a “No Because” sign was used as part of 

the norms to discourage participants from discussing why the data might look the way they do). After reviewing the data, the 
team then made inferences and identified additional questions it might need to answer. Finally, the team discussed implications 
for the school practices as well as next steps. Although the data coaches worked with all schools to perform these data dives, 

it will take time for the dives to become entrenched as part of the school culture. The work will continue to reach additional 
educators in each school. 


Phase 3- Infer/ Question 
Generate possible explanations 


What inferences and explanations might we draw? 
What questions are we asking? 
What tentative conclusions might we draw? 


Adapted from Laura Lipton and Bruce Wellman 


@rancy Lowe, TERY 


Figure 11. Sample data dive protocol. 


In addition to data dives, the data coaches developed reports for the administration. Part of the initial goal of this effort was 
to create consistency across the district and to look at core instruction across the district. Analysis of district-wide data led to 
meaningful discussions that helped guide school and district decisions. For example, school and district officials realized the 
need to focus on core math fluency at the earlier elementary levels and the need for upper elementary teachers to be open 

to review and remediation (see figure 12, next page). At the middle school level, a Professional Learning Community (PLC) 
system was used to look at writing skills for all sixth- through eighth-grade students. Teams of teachers from all disciplines 
looked at writing prompts and rubrics, dissected sample student writing pieces, and began to understand the value of a team 
approach to understanding student writing skills based on standards-aligned assessments. 


Much work is still needed to ensure that administrators and educators use the data effectively. The infrastructure is in place for 
most schools but must still be expanded to all schools. Collectively, the analysis is expected to improve student outcomes, and 
review will continue in future years. 
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Through the process of collecting and analyzing the data, Knowledge, Skills and Professional Behaviors were critical to success. 


Knowledge. This phase of work included collection (K.2.C Data Collection) of the defined measurements (K.2. C Data 
Collection; K.1.E Data Metric). Rubrics were defined and reviewed to collect quantitative data (K.7.C Types of Data). The 

data format for the assessments was defined to ensure that data were collected as intended (K.2.E Data Format). 'The data 
collection steps were clearly defined (K.2.C Data Collection). Educators were provided the data context so that they would 
understand why the data were being collected (K.2.D Data Context). The team discussed and determined what data privacy 
measures should be in place for both the implementation documents and student data (K.3.D Privacy and Security). The team 
recognized the limitations of both the existing data and the data that would be collected (K.3.B Data Limitations). 


Skills. During this phase, there was very little time between planning and collecting the data (8.7 Planning; S.3 Collecting). 

The data management (S.7.C Data Management) process was defined just in time to develop the measures (S.2.C Development 
of Measures). The measures had to consider what scores would be captured for each student so that valuable interpretations 
could be made to influence instruction (S.2.C Development of Measures). Initial data dives demonstrated how individuals 

can often misrepresent the understanding of data; teams would first describe what they thought the data would tell them, 
but when analyzing the data they found they were often mistaking student outcomes when viewing the actual results (8.5.B 
Representation). The district data team and school-level teams have begun creating strategies and action plans based on the findings in 
the data to improve student outcomes (S.7.A Strategies; S.7.B Action Plan). 


Professional Behaviors. Work must continue, but the foundation is in place to entrench a culture and climate that enables 
the collaborative use of data (B.7.E Culture; B.3.B Collaborative Climate; B.3.A Collaborative Use). Professional development 
must be expanded to ensure that teachers have a personal ethical commitment to prioritizing the use of data (B.4.C 
Professional Development; B.1.D Ethics). The PerformancePLUS system was used in part because it provides the security to meet 
the rules and laws to protect students (B.2.4 Rules and Laws; B.2.B Protection). As the district team and data coaches proceed 
with continuous improvement in mind (B.4 Continuous Improvement), a focus will drive problem solving by using data to 
improve instruction (B.4.A Problem Solving). 
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Figure 12. Sample data analysis reports: by grade, individual results. 
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LESSONS LEARNED: THE FINAL LANDING 


This work resulted in significant contributions and achievements. It was driven by top district administrators and executed 
by teachers, specialists, and principals working alongside data coaches. However, more work is needed to address gaps in the 
initial implementation. Initial challenges included setting up the system of administering new assessments and collecting the data. 


Communication 


The need for consistent and early communication should be a focus for the ongoing work (S.6 Communicating). Teachers 
needed more time to process and understand the data collection (assessment) requirements to ensure consistent collaboration 
(K.2.C Data Collection; B.3. Collaboration). Communication in a school district this large was inconsistent. No generally accepted 
processes were in place to keep principals and teachers informed of changes. Multiple versions of documents were sometimes 
sent to schools, which created data inconsistency and added to the confusion (K.7.B Data Quality). Disseminating information 
was a bit easier at the elementary level than at the middle school level. 


All the teachers should have a clear understanding of the questions being addressed (K.7.A Question Formation), the types of 
data and measurements being collected (K.7.C Types of Data; K.1.D Types of Measures), the meaning of the data (K.7.E Data 
Metric), the need to focus on data quality (K.7.B Data Quality), and the data analysis tools (K.2.B Data Analysis Tools). The focus 
from the beginning should include a discussion about how the collected data will provide value to the educators, as well as how 
they can and should use the data to inform instruction. 


Metrics Review 


The full district administrative team should review and analyze the data on a regular basis to understand the findings, ensure 
full compliance, discuss ways to tweak the implementation, and implement instructional change for students. The full 
administrative team—including all principals—must have increased professional development to be able to critically evaluate 
the data at monthly team meetings (B.4.C Professional Development; S.2.B Critical Evaluation). The work data coaches have 
performed to format and compare the data (S.4.A Formatting; S.4.E. Comparisons) should be done by the administrators. 


Time 

It is vital that leaders of this initiative be provided with the dedicated time to complete the work; it should be a clear priority 
with an emphasis on outreach (B.3.D Prioritization; B.3.C Outreach). For example, seven of the 14 elementary schools had 
math coaches. Schools with assigned coaches were more informed than those without coaches and therefore more successful 
in planning, analyzing, and interpreting the work (S.7 Planning; 8.4 Analyzing; S.5 Interpreting). Information was to go through 
ptincipals for un-coached schools, but no system was established to communicate on a regular basis (S.6 Communicating). 
Additionally, at the middle school level, classroom teachers rather than math coaches were driving this work. The teachers had 
very little time to work on the assessments, to use data to solve problems (B.4.A Problem Solving), ot to continually improve the 
implementation structure (B.4.B Improving Outcomes). Over time, all principals and math teachers received information, but ensuring 
more dedicated time for a lead in each school would have allowed for more consistency and ownership at the school level. 


Professional Development 


The infrastructure is in place to solidify a collaborative data culture in Manchester (B.7.E Cu/ture). The fundamental knowledge 
for identifying the types of measurements (K.7.D Types of Measures), the data collection (K.2.C Data Collection), and the 

analysis tools (K.2.B Data Analysis Tools) are in place. Critical evaluation has begun (S.2.B Critical Evaluation); reports have 

been formatted (K.2.E Data Format); and patterns have been identified by those most involved in the analysis effort (S.5.C 
Patterns). Although the data are being collected at most schools for the vast majority of students, the data are being entered 
inconsistently across the schools, which indicates a need for improved high-quality data (K.7.B Data Quality). 


The schools led by principals with experience using data to inform instruction are faithful to the process. They understand 
that it takes time, and they trust in the process. The schools with math coaches on staff are also moving forward with data 
review and analysis. But to develop a system culture (B.7.E Cu/ture), the district must increase professional development (B.4.C 
Professional Development) to continuously improve (Professional Behaviors, Continuous Improvement) and ensure that all teachers can 
solve problems to improve instruction (B.4.A Problem Solving). Teacher teams must have the time to focus on learning the system 
ptactices that have been put in place. School-level and district-level meetings must increase their discussion of these practices and 
of the value inherent in this analysis, and professional development should highlight the positive impact on student performance. 
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SUMMARY 


The New Hampshire SLDS team hopes to continue working with the Manchester School District. The district has made huge 

gains in the past year for its students and for its teachers. At the beginning of 2014, teachers were not following common units 
of study, they did not have common assessments that aligned with their standards, they had very little data, and—tfor the most 
part—teachers did not have a formal data analysis and review process to inform and improve their instruction. 


By the end of the 2014—2015 school year, common units of study will be in place, common assessments for all students in 
kindergarten through grade 12 will be used, and data will be collected based on those assessments. The process of analyzing 
and using the data will have begun. 


During the 2015—2016 school year, officials hope to continue the assessment and data collection process and make large 
sttides toward building a collaborative culture in which all teachers are engaged with using data and improving student learning, 
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Appendix A: Data Use Standards 


KNOWLEDGE 


K.1 Fundamental 


K.1.A Question Formation 
K.1.B Data Quality 

K.1.C Types of Data 

K.1.D Types of Measures 
K.1.E Data Metric 

K.1.F Data Sources 

K.1.G Data Representations 


K.2 Processing 


K.2.A Types of Analysis 
K.2.B Data Analysis Tools 
K.2.C Data Collection 
K.2.D Data Context 
K.2.E Data Format 


K.3 Considerations 


K.3.A Data Assumptions 
K.3.B Data Limitations 
K.3.C Data Culture 

K.3.D Privacy and Security 
K.3.E Data Ethics 


SKILLS 


S.1 Planning 


S.1.A Goals and Questions 
S.1.B Alignment 

S.1.C Data Management 
S.1.D Data Meaning 


S.2 Selecting 


S.2.A Data Discovery and Data Acquisition 
S.2.B Critical Evaluation 
S.2.C Development of Measures 


S.3 Collecting 


S.3.A Facilitation 

S.3.B Technology 

$.3.C Multiple Measures 
S.3.D Modifications 


S.4 Analyzing 


S.4.A Formatting 

S.4.B Data Cleaning 
S.4.C Aligned Analysis 
S.4.D Considerations 
S.4.E Comparisons 
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S.5 Interpreting 


S.5.A Locating 

S.5.B Representation 
S.5.C Patterns 

S.5.D Congruency 


$.6 Communicating 


S.6.A Presentation 
S.6.B Explanation 
S.6.C Multiple Audiences 


S.7 Acting 


S.7.A Strategies 
S.7.B Action Plan 


PROFESSIONAL BEHAVIORS 


B.1 Ethical Use 


B.1.A Data Quality 
B.1.B Transparency 
B.1.C Representation 
B.1.D Ethics 

B.1.E Culture 

B.1.F Use 


B.2 Rules and Regulations 


B.2.A Rules and Laws 
B.2.B Protection 
B.2.C Advocacy for Protections 


B.3 Collaboration 


B.3.A Collaborative Use 
B.3.B Collaborative Climate 
B.3.C Outreach 

B.3.D Prioritization 


B.4 Continuous Improvement 


B.4.A Problem Solving 
B.4.B Improving Outcomes 
B.4.C Professional Development 
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Workgroup Members 


Po State | Nome | gcnization 


California Christina Tydeman 
Guam Zeni Natividad 
Hawaii Justin Katahira? 
Hawdii Christine Sorensen Irvine 
Idaho Andy Mehl 

Illinois Doug Franklin 
Kansas Eric Punswick 
Kansas Lori Adams 
Montana Brett Carter 
Nebraska Russ Masco? 
Nebraska Dick Meyers 


Irene Koffink 

Mike Schwartz 
David Weinberger 
Nathan Anderson 
Magdalena Brockel 
Stacy Duffield 

Amy Engelhard 

Jen Glasheen 


New Hampshire 
New Hampshire 
New York 

North Dakota 
North Dakota 
North Dakota 
North Dakota 
North Dakota 


Dominican University of California 
Guam Department of Education 


Hawaii P-20 Partnerships for Education 
Pacific Regional Comprehensive Center; University 


of Hawaii at Manoa 

Idaho Board of Higher Education 
Illinois Board of Higher Education 
Olathe Public Schools 


Kansas State Department of Education 


Montana Office of Public Instructi 


on 


Nebraska Department of Education 
University of Nebraska at Kearney, 


College of Education 


New Hampshire Department of Education 
New Hampshire Department of Education 


Yonkers Public Schools 


Mid-Dakota Education Cooperative 


Missouri River Educational Coope 
North Dakota State University 
Education Technology Council 


rative 


Southeast Education Cooperative 


North Dakota Steve Snow North Dakota Department of Instruction 
Oklahoma Bryan Duke University of Central Oklahoma 

Oklahoma James Machell University of Central Oklahoma 

Oklahoma Susan Pinson Oklahoma State Department of Education 
South Dakota Sara Kock?* South Dakota Department of Education 
Tennessee Margie Johnson® Metro Nashville Public Schools 

Washington Melissa Beard Education Research and Data Center 
Washington Jim DePaepe Central Washington University 

Facilitator Corey Chatis SLDS Grant Program State Support Team 


? Professional Behaviors Subgroup Lead 
3 Skills Subgroup Co-Lead 

* Knowledge Subgroup Lead 

° Case Studies Subgroup Lead 
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